{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "69ad077f-af4e-49b8-b549-fb3112299aa8",
   "metadata": {
    "id": "69ad077f-af4e-49b8-b549-fb3112299aa8"
   },
   "source": [
    "# **LLM Zoomcamp 2024 Competition Starter Notebook**\n",
    "\n",
    "Welcome to the starter notebook for the [this Kaggle competition](https://www.kaggle.com/competitions/llm-zoomcamp-2024-competition/overview) organized by [DataTalksClub](https://datatalks.club/) as part of the course [LLM Zoomcamp 2024](https://github.com/DataTalksClub/llm-zoomcamp).\n",
    "\n",
    "In this competition, the challenge is to solve high school math questions using Large Language Models.\n",
    "\n",
    "\n",
    "\n",
    "![image.png]()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c86ebbeb-c070-45d4-99ef-de33b53d447d",
   "metadata": {
    "id": "c86ebbeb-c070-45d4-99ef-de33b53d447d"
   },
   "source": [
    "### Install and Import Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "v5gOKYkO3sI2",
   "metadata": {
    "id": "v5gOKYkO3sI2"
   },
   "outputs": [],
   "source": [
    "# Install necessary packages\n",
    "! pip install -qU langchain-openai langchain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f7a0c7bd-512d-4e5b-ab43-1bc3b8c97fd4",
   "metadata": {
    "id": "f7a0c7bd-512d-4e5b-ab43-1bc3b8c97fd4"
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90E3nLng33Ct",
   "metadata": {
    "id": "90E3nLng33Ct"
   },
   "source": [
    "### Set up OpenAI API key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "iPETn3V83rc3",
   "metadata": {
    "id": "iPETn3V83rc3"
   },
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "OPENAI_API_KEY:  ········\n"
     ]
    }
   ],
   "source": [
    "import getpass\n",
    "import os\n",
    "\n",
    "def _get_env(var: str):\n",
    "    if not os.environ.get(var):\n",
    "        os.environ[var] = getpass.getpass(f\"{var}: \")\n",
    "\n",
    "_get_env(\"OPENAI_API_KEY\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e4ad035-2c1a-4e31-93b4-58793d219bc9",
   "metadata": {
    "id": "2e4ad035-2c1a-4e31-93b4-58793d219bc9"
   },
   "source": [
    "### Application Design\n",
    "\n",
    "Some helper functions to set up this app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "MD8YN6JYCT0J",
   "metadata": {
    "id": "MD8YN6JYCT0J"
   },
   "outputs": [],
   "source": [
    "from langchain_openai import OpenAI\n",
    "from langchain.prompts import PromptTemplate\n",
    "\n",
    "llm = OpenAI(model_name=\"gpt-3.5-turbo-instruct\", temperature=0, max_tokens=1500)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tuOo27j1SDl3",
   "metadata": {
    "id": "tuOo27j1SDl3"
   },
   "source": [
    "#### Function to query the LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "Ofi_fPrGPHLd",
   "metadata": {
    "id": "Ofi_fPrGPHLd"
   },
   "outputs": [],
   "source": [
    "def get_answer(question):\n",
    "    prompt = f\"\"\"Please solve the following high school math problem step by step. Explain your reasoning clearly and provide the final answer.\n",
    "\n",
    "{question}\n",
    "\n",
    "Step-by-step solution and final answer:\"\"\"\n",
    "\n",
    "    response = llm.invoke(prompt)\n",
    "    return response.strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sQoV28mwPMcY",
   "metadata": {
    "id": "sQoV28mwPMcY"
   },
   "source": [
    "#### Function to Extract Answer with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "35CWYSJgPVNe",
   "metadata": {
    "id": "35CWYSJgPVNe"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def extract_numerical_answer(text):\n",
    "    # Look for patterns like \"Final answer: X\" or \"The answer is X\" at the end of the text\n",
    "    match = re.search(r'(?:final answer|the answer is)[:\\s]*([+-]?\\d*\\.?\\d+)', text, re.IGNORECASE)\n",
    "    if match:\n",
    "        return float(match.group(1))\n",
    "    else:\n",
    "        # If no clear final answer, look for the last number in the text\n",
    "        numbers = re.findall(r'[+-]?\\d*\\.?\\d+', text)\n",
    "        return float(numbers[-1]) if numbers else None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qeLhlhiTPm4I",
   "metadata": {
    "id": "qeLhlhiTPm4I"
   },
   "source": [
    "#### Function to Prepare Dataset and Get Answers with"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbca849b-d271-4a9e-bc6e-1ab47af1bffe",
   "metadata": {
    "id": "qeLhlhiTPm4I"
   },
   "source": [
    "We will use a pool executor to make it faster (`max_workers=6` controls how many concurrent queries we send to OpenAI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "17e8befe-4f9e-4342-8945-54b96a4c1678",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm\n",
    "\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "\n",
    "pool = ThreadPoolExecutor(max_workers=6)\n",
    "\n",
    "def map_progress(pool, seq, f):\n",
    "    results = []\n",
    "\n",
    "    with tqdm(total=len(seq)) as progress:\n",
    "        futures = []\n",
    "\n",
    "        for el in seq:\n",
    "            future = pool.submit(f, el)\n",
    "            future.add_done_callback(lambda p: progress.update())\n",
    "            futures.append(future)\n",
    "\n",
    "        for future in futures:\n",
    "            result = future.result()\n",
    "            results.append(result)\n",
    "\n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc3d5c12-6c58-4d45-b8f5-299f84c1fbe3",
   "metadata": {},
   "source": [
    "Now we create a function for processing each row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e620d41d-8298-47f1-9a14-a257727157f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_row(row):\n",
    "    problem_id = row['problem_id']\n",
    "    problem_text = row['problem_text']\n",
    "\n",
    "    llm_reasoning = get_answer(problem_text)\n",
    "\n",
    "    numerical_answer = extract_numerical_answer(llm_reasoning)\n",
    "\n",
    "    return {\n",
    "        'problem_id': problem_id,\n",
    "        'problem_text': problem_text,\n",
    "        'llm_reasoning': llm_reasoning,\n",
    "        'answer': numerical_answer\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eb90eaf-82aa-422a-85de-dd3769ac9723",
   "metadata": {},
   "source": [
    "At this point, let's load the train data - so we can use it for evaluation our solution offline (without submitting to Kaggle):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "DimbAMWpP6Dj",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "DimbAMWpP6Dj",
    "outputId": "693e5a26-657f-4edc-a693-57d8e16dcfe2"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>problem_id</th>\n",
       "      <th>problem_text</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2374</td>\n",
       "      <td>Find the value of the expression $\\dfrac{17}{5...</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4723</td>\n",
       "      <td>In a company of 30 people, 25 use the social n...</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7135</td>\n",
       "      <td>The number of road traffic accidents (RTAs) in...</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5814</td>\n",
       "      <td>Find the value of the expression $\\dfrac{2\\str...</td>\n",
       "      <td>256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9237</td>\n",
       "      <td>A traveler from Moscow wants to visit four cit...</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   problem_id                                       problem_text answer\n",
       "0        2374  Find the value of the expression $\\dfrac{17}{5...    1.6\n",
       "1        4723  In a company of 30 people, 25 use the social n...     24\n",
       "2        7135  The number of road traffic accidents (RTAs) in...     32\n",
       "3        5814  Find the value of the expression $\\dfrac{2\\str...    256\n",
       "4        9237  A traveler from Moscow wants to visit four cit...     53"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train = pd.read_csv('data/train.csv')\n",
    "df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a694391e-a890-4507-816c-5cf118e90bce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'problem_id': 2374,\n",
       " 'problem_text': 'Find the value of the expression $\\\\dfrac{17}{5} :\\\\dfrac{34}{3} +1.3$.',\n",
       " 'llm_reasoning': 'To solve this problem, we will first simplify the expression $\\\\dfrac{17}{5} :\\\\dfrac{34}{3}$.\\n\\nTo divide fractions, we need to invert the second fraction and multiply. This can be written as:\\n\\n$\\\\dfrac{17}{5} :\\\\dfrac{34}{3} = \\\\dfrac{17}{5} \\\\times \\\\dfrac{3}{34}$\\n\\nNext, we can simplify the fractions by canceling out any common factors. In this case, we can divide both the numerator and denominator of the first fraction by 17, and the second fraction by 2. This gives us:\\n\\n$\\\\dfrac{1}{5} \\\\times \\\\dfrac{3}{17} = \\\\dfrac{3}{85}$\\n\\nNow, we can substitute this simplified fraction back into the original expression:\\n\\n$\\\\dfrac{17}{5} :\\\\dfrac{34}{3} +1.3 = \\\\dfrac{3}{85} + 1.3$\\n\\nTo add fractions, we need to have a common denominator. In this case, we can multiply the first fraction by $\\\\dfrac{17}{17}$ to get a common denominator of 85. This gives us:\\n\\n$\\\\dfrac{3}{85} + 1.3 = \\\\dfrac{3}{85} + \\\\dfrac{85}{85} \\\\times 1.3 = \\\\dfrac{3}{85} + \\\\dfrac{110.5}{85}$\\n\\nNow, we can add the fractions together:\\n\\n$\\\\dfrac{3}{85} + \\\\dfrac{110.5}{85} = \\\\dfrac{113.5}{85}$\\n\\nFinally, we can simplify the fraction by dividing both the numerator and denominator by 5:\\n\\n$\\\\dfrac{113.5}{85} = \\\\dfrac{22.7}{17}$\\n\\nTherefore, the final answer is $\\\\dfrac{22.7}{17}$.',\n",
       " 'answer': 17.0}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rows = df_train.head().to_dict(orient='records')\n",
    "process_row(rows[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a3da223-3af9-4703-a239-b103d4e5575e",
   "metadata": {},
   "source": [
    "This is how we combine the two:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "67df87cf-e27d-45bc-94fa-16d1e543b086",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2cd40362c0524b8ca30c1e4ed3496767",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>problem_id</th>\n",
       "      <th>problem_text</th>\n",
       "      <th>llm_reasoning</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2374</td>\n",
       "      <td>Find the value of the expression $\\dfrac{17}{5...</td>\n",
       "      <td>To solve this problem, we will first simplify ...</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4723</td>\n",
       "      <td>In a company of 30 people, 25 use the social n...</td>\n",
       "      <td>1) In this company, there will be 10 people wh...</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7135</td>\n",
       "      <td>The number of road traffic accidents (RTAs) in...</td>\n",
       "      <td>Step 1: Let x be the number of RTAs in the win...</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5814</td>\n",
       "      <td>Find the value of the expression $\\dfrac{2\\str...</td>\n",
       "      <td>To solve this problem, we will use the propert...</td>\n",
       "      <td>256.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9237</td>\n",
       "      <td>A traveler from Moscow wants to visit four cit...</td>\n",
       "      <td>To visit all four cities and spend less than 5...</td>\n",
       "      <td>46.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   problem_id                                       problem_text  \\\n",
       "0        2374  Find the value of the expression $\\dfrac{17}{5...   \n",
       "1        4723  In a company of 30 people, 25 use the social n...   \n",
       "2        7135  The number of road traffic accidents (RTAs) in...   \n",
       "3        5814  Find the value of the expression $\\dfrac{2\\str...   \n",
       "4        9237  A traveler from Moscow wants to visit four cit...   \n",
       "\n",
       "                                       llm_reasoning  answer  \n",
       "0  To solve this problem, we will first simplify ...    17.0  \n",
       "1  1) In this company, there will be 10 people wh...    24.0  \n",
       "2  Step 1: Let x be the number of RTAs in the win...    32.0  \n",
       "3  To solve this problem, we will use the propert...   256.0  \n",
       "4  To visit all four cities and spend less than 5...    46.0  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = map_progress(pool, rows, process_row)\n",
    "df_results = pd.DataFrame(results)\n",
    "df_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4b5ae96e-31dc-49f3-a1eb-39497d235cd0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>problem_id</th>\n",
       "      <th>problem_text</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2374</td>\n",
       "      <td>Find the value of the expression $\\dfrac{17}{5...</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4723</td>\n",
       "      <td>In a company of 30 people, 25 use the social n...</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7135</td>\n",
       "      <td>The number of road traffic accidents (RTAs) in...</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5814</td>\n",
       "      <td>Find the value of the expression $\\dfrac{2\\str...</td>\n",
       "      <td>256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9237</td>\n",
       "      <td>A traveler from Moscow wants to visit four cit...</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   problem_id                                       problem_text answer\n",
       "0        2374  Find the value of the expression $\\dfrac{17}{5...    1.6\n",
       "1        4723  In a company of 30 people, 25 use the social n...     24\n",
       "2        7135  The number of road traffic accidents (RTAs) in...     32\n",
       "3        5814  Find the value of the expression $\\dfrac{2\\str...    256\n",
       "4        9237  A traveler from Moscow wants to visit four cit...     53"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a91fdfb6-ee87-401f-bdef-f9628d7fe4f0",
   "metadata": {},
   "source": [
    "Let's evaluate the answer on these 5 examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ff311f15-0302-48cb-8569-af52f8b664a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scorer import score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "491bce46-7d0c-49fd-a1cd-5d2cb85d4f83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "score(df_train.head(), df_results, 'problem_id')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa342dc1-45b6-419c-91a2-dc6fb3c59490",
   "metadata": {},
   "source": [
    "So, 3/5 correct answers. Note that the results are not always deterministic, so the quality may vary.\n",
    "\n",
    "Let's now put this together in a function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "gVrSUBPfPrzk",
   "metadata": {
    "id": "gVrSUBPfPrzk"
   },
   "outputs": [],
   "source": [
    "def prepare_prompts_and_get_answers(df):\n",
    "    rows = df.to_dict(orient='records')\n",
    "    results = map_progress(pool, rows, process_row)\n",
    "    return pd.DataFrame(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8AnFw9YaSjyJ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8AnFw9YaSjyJ",
    "outputId": "31fa3be5-98fb-499f-d2cb-0b35f814f1e3"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a29955a174bd4ccfbea52310b42f4c9f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>problem_id</th>\n",
       "      <th>problem_text</th>\n",
       "      <th>llm_reasoning</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2374</td>\n",
       "      <td>Find the value of the expression $\\dfrac{17}{5...</td>\n",
       "      <td>To solve this problem, we will first simplify ...</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4723</td>\n",
       "      <td>In a company of 30 people, 25 use the social n...</td>\n",
       "      <td>1) In this company, there will be 10 people wh...</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7135</td>\n",
       "      <td>The number of road traffic accidents (RTAs) in...</td>\n",
       "      <td>Step 1: Let x be the number of RTAs in the win...</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5814</td>\n",
       "      <td>Find the value of the expression $\\dfrac{2\\str...</td>\n",
       "      <td>To solve this problem, we will use the propert...</td>\n",
       "      <td>256.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9237</td>\n",
       "      <td>A traveler from Moscow wants to visit four cit...</td>\n",
       "      <td>To visit all four cities and spend less than 5...</td>\n",
       "      <td>46.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   problem_id                                       problem_text  \\\n",
       "0        2374  Find the value of the expression $\\dfrac{17}{5...   \n",
       "1        4723  In a company of 30 people, 25 use the social n...   \n",
       "2        7135  The number of road traffic accidents (RTAs) in...   \n",
       "3        5814  Find the value of the expression $\\dfrac{2\\str...   \n",
       "4        9237  A traveler from Moscow wants to visit four cit...   \n",
       "\n",
       "                                       llm_reasoning  answer  \n",
       "0  To solve this problem, we will first simplify ...     1.6  \n",
       "1  1) In this company, there will be 10 people wh...    24.0  \n",
       "2  Step 1: Let x be the number of RTAs in the win...    32.0  \n",
       "3  To solve this problem, we will use the propert...   256.0  \n",
       "4  To visit all four cities and spend less than 5...    46.0  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_results = prepare_prompts_and_get_answers(df_train)\n",
    "df_train_results.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f8983d04-d11c-4052-bcb4-09c59b5d77f0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.42"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "score(df_train, df_train_results, 'problem_id')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80b0f207-be0d-4a6d-aafd-0e1b3d132279",
   "metadata": {},
   "source": [
    "So in our tests, the solution is 42% correct. Let's prepare a test submission"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2e6e020-0fdf-413c-9779-20ef99725ee3",
   "metadata": {},
   "source": [
    "## Preparing the test submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "WV8oslsxRUhX",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WV8oslsxRUhX",
    "outputId": "e363b602-9bd8-44b1-a8f6-ab2df1b4bbbb"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba2d07b1e52d46fa80d3fdab594ef736",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_test = pd.read_csv('data/test.csv')\n",
    "\n",
    "df_test_results = prepare_prompts_and_get_answers(df_test)\n",
    "\n",
    "submission = df_test_results[['problem_id', 'answer']]\n",
    "submission.to_csv('starter_notebook_submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77d61c39-e01c-4192-9467-5820602e6ee7",
   "metadata": {},
   "source": [
    "Now submit it to Kaggle. This solution gets 50% on the public leaderboard"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6H16KMkESocU",
   "metadata": {
    "id": "6H16KMkESocU"
   },
   "source": [
    "### **Conclusion and Tips for Success**\n",
    "\n",
    "Congratulations on getting started with this competition! As you proceed, keep these valuable insights in mind:\n",
    "\n",
    "* **Understanding Langchain**\n",
    "\n",
    "  Imagine foundation models (like ChatGPT, Claude, etc.) as ingredients, and companies ([OpenAI](https://platform.openai.com/playground/chat), [Anthropic](https://console.anthropic.com/workbench), [Cohere](https://cohere.com/) as vendors. In this analogy, [Langchain](https://python.langchain.com/v0.2/docs/introduction/) is your kitchen - a place to combine these ingredients from different vendors into a finished dish (your project). While this starter uses Langchain, explore other \"kitchens\" like [LlamaIndex](https://www.llamaindex.ai/) or [FlowiseAI](https://docs.flowiseai.com/) to find the best tools for your specific needs.\n",
    "  \n",
    "* **Model Selection**\n",
    "\n",
    "  The solution provided uses GPT-3.5-turbo-instruct, a model fine-tuned for following instructions. This choice is deliberate, given the question-answer format of the dataset. When selecting models, consider their strengths in relation to your task.\n",
    "\n",
    "* **Evaluate Model Performance**\n",
    "\n",
    "  Don't just pick a model blindly. Research LLM leaderboards (like the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard)) to understand how different models perform across various metrics. This information can guide your model selection process.\n",
    "\n",
    "* **Explore Paid Options**\n",
    "\n",
    "  While the baseline solution uses a free model, don't limit yourself. Paid models like GPT-4 often outperform free alternatives. Consider the potential return on investment when deciding whether to use paid services.\n",
    "\n",
    "* **Master the Art of Prompting**\n",
    "\n",
    "  You're the chef in this AI kitchen. Experiment with different prompts to guide the LLM's responses. Be specific in your instructions, and think creatively about how to elicit the best possible answers from the model. For example, *You are a world-class expert at solving High school olympiad math questions...*\n",
    "\n",
    "* **Iterate and Improve**\n",
    "\n",
    "  Use the baseline as a starting point, not an end goal. Continuously refine your approach, trying different models, prompts, and post-processing techniques to improve your results.\n",
    "\n",
    "* **Stay Curious and Adaptive**\n",
    "\n",
    "  The field of AI is rapidly evolving. Stay open to new models, techniques, and tools that emerge during the competition. Your ability to adapt and incorporate new ideas could be the key to success. For example: Recently, Anthropic released a new model that is said to be better than GPT-4o at reasoning tasks.\n",
    "\n",
    "Remember, this competition is not just about reaching a high score – it's an opportunity to deepen your understanding of LLMs and develop practical skills in AI application development. Enjoy the process of learning and discovery as you work towards your solution. Good luck!\n",
    "\n",
    "Okay, it's time to cook, chef!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02ea28c5-1042-4b98-9219-498ec56f597f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
